Experiment 1a

Contingencies


Low Value Options
  • Fixed: 20 (Door 3)
  • Risky: 0 or 40 (Door 4)
High Value Options
  • Fixed: 60 (Door 1)
  • Risky: 40 or 80 (Door 2)
Trials
  • Six blocks of 80 trials.
  • Decision Trials: (fixed low vs risky low) & (fixed high vs risky high) - 24 per block (12 low, 12 high)
  • Single door trials: one door to click - 40 per block
  • Catch trials: high vs low - 16 per block
EX 80-20
  • Block 1: 80% extreme values (0 low, 80 high)
  • Block 2: 50% extreme values.
  • Block 3: 20% extreme values (0 low, 80 high)
  • Block 4-6: Block 2: 50% extreme values.
EX 20-80
  • Block 1: 20% extreme values (0 low, 80 high)
  • Block 2: 50% extreme values.
  • Block 3: 80% extreme values (0 low, 80 high)
  • Block 4-6: 50% extreme values.
EX 50-50 (control)
  • All blocks: 50% extreme values.


Participant Information

Pre-catch exclusion

N = 192

group gender n age_mean age_sd age_median age_IQR
EX 50-50 F 38 19.132 1.711 18 1.75
EX 50-50 M 25 20.320 3.198 19 3.00
EX 80-20 F 38 19.447 3.644 18 1.75
EX 80-20 M 28 19.607 2.347 19 2.00
EX 20-80 F 39 20.179 4.667 19 2.00
EX 20-80 M 24 19.375 1.952 19 2.00


Post-catch exclusion

N = 184

group gender n age_mean age_sd age_median age_IQR
EX 50-50 F 36 19.194 1.737 18.5 2
EX 50-50 M 22 20.318 3.272 19.0 3
EX 80-20 F 37 19.378 3.669 18.0 1
EX 80-20 M 27 19.519 2.343 19.0 2
EX 20-80 F 39 20.179 4.667 19.0 2
EX 20-80 M 23 19.435 1.973 19.0 2


Condition totals:

group n age_mean age_sd age_median age_IQR
EX 50-50 58 19.621 2.470 19 2
EX 80-20 64 19.438 3.157 19 2
EX 20-80 62 19.903 3.887 19 2


  • Total number of catch trial exclusions = 8.

  • Catch trial exclusions were based on block 5 & 6 performance.




Catch Trials

Plot - Block means with 95% CIs





Risk Preference Trials

Plot - Block means with 95% CIs



Plot - Interaction plot with 95% CIs (Block 5 & 6)



Main Effects

ANOVA Table

Model df AIC BIC logLik Test L.Ratio p.value BF_10
base_mod 1 4 226.496 242.128 -109.248 NA NA NA
grp_mod 2 6 228.323 251.772 -108.162 1 vs 2 2.172 0.338 8.000000e-03
value_mod 3 7 176.484 203.840 -81.242 2 vs 3 53.840 0.000 2.559852e+10
grp_val 4 9 179.296 214.469 -80.648 3 vs 4 1.188 0.552 5.000000e-03

Main Effect of Group:

\(\chi^2(2) = 2.17, p = 0.338, BF_{10} < .01\)

Main Effect of Choice Value (i.e., Extreme Outcome effect):

\(\chi^2(1) = 53.84, p < .001, BF_{10} > 150\)

Interaction: Group \(\times\) Choice Value:

\(\chi^2(2) = 1.19, p = 0.552, BF_{10} < .01\)


Planned Comparisons

ANOVA Table

Value Std.Error DF t-value p-value sig r_effect
(Intercept) 0.413 0.034 181 12.219 0.000 TRUE 0.672
group 8020_vs_50 0.041 0.047 181 0.872 0.384 FALSE 0.065
group 2080_vs_50 -0.026 0.047 181 -0.552 0.581 FALSE 0.041
choice_value1 0.102 0.024 181 4.336 0.000 TRUE 0.307
group 8020_vs_50 :choice_value1 -0.014 0.032 181 -0.435 0.664 FALSE 0.032
group 2080_vs_50 :choice_value1 0.020 0.033 181 0.620 0.536 FALSE 0.046

EX 80-20 vs. EX 50-50:

\(b = 0.041, t(181) = 0.872, p = 0.384, r = 0.065\)

EX 20-80 vs. EX 50-50:

\(b = -0.026, t(181) = -0.552, p = 0.581, r = 0.041\)

EX 80-20 vs. EX 50-50 | Choice Value (High vs. Low):

\(b = -0.014, t(181) = -0.435, p = 0.664, r = 0.032\)

EX 20-80 vs. EX 50-50 | Choice Value (High vs. Low):

\(b = 0.02, t(181) = 0.62, p = 0.536, r = 0.046\)

Notes
  • All plots and analyses are showing the combined effect of block 5 and 6.
  • Bayes Factors are reported as Inverse Bayes Factors, this means larger values indicate greater support for the effect. Anything less than 1 should be considered as contributing no evidence.




First-Outcome Judgement

Plot - Proportions


Note:

  • Proportions are taken within group and context (High/Low). E.g., All three bars in the top left facet sum to 1, all three bars in the top middle facet sum to 1, and so on.

  • Since non-numeric responses were permissible by the experiment’s program, data was filtered to remove all non-numeric inputs. If a participant’s inputs had a cumulative sum less than 20 or greater than 300, their results were removed. In total, 8 participants met this criteria.


2x3 Chi-Squared Tests

High Value Results

High Value Contingency Table
No Yes
EX 50-50 11 38
EX 80-20 13 44
EX 20-80 16 39


\(\chi^2(2) = 0.81, p = 0.667, \varphi_c = 0.07\)

  • Cannot rule out that recalling the extreme value is independent of group.
Standardized Residuals
No Yes
EX 50-50 -0.465 0.465
EX 80-20 -0.443 0.443
EX 20-80 0.898 -0.898
p-values
No Yes
EX 50-50 0.642 0.642
EX 80-20 0.658 0.658
EX 20-80 0.369 0.369








None of the three conditions differ significantly from the expected value.


Low Value Results

Low Value Contingency Table
No Yes
EX 50-50 12 37
EX 80-20 8 49
EX 20-80 7 48


\(\chi^2(2) = 3.04, p = 0.219, \varphi_c = 0.14\)

  • Cannot rule out that recalling the extreme value is independent of group.
Standardized Residuals
No Yes
EX 50-50 1.734 -1.734
EX 80-20 -0.688 0.688
EX 20-80 -0.989 0.989
p-values
No Yes
EX 50-50 0.083 0.083
EX 80-20 0.492 0.492
EX 20-80 0.323 0.323








  • None of the three conditions differ significantly from the expected value.


Post-Hoc Pairwise Fisher-Exact Tests

value comparison odds_ratio p_value CI_low CI_upper p_value_adj
High EX 50-50 : EX 80-20 0.9799 1.0000 0.3521 2.6883 1.0000
High EX 50-50 : EX 20-80 0.7080 0.5056 0.2603 1.8687 1.0000
High EX 80-20 : EX 20-80 0.7223 0.5201 0.2805 1.8310 1.0000
Low EX 50-50 : EX 80-20 1.9735 0.2157 0.6633 6.1822 1.0000
Low EX 50-50 : EX 20-80 2.2068 0.1357 0.7174 7.3155 0.8141
Low EX 80-20 : EX 20-80 1.1184 1.0000 0.3256 3.9337 1.0000


  • Nothing reached significance.


Note:

  • In the contingency tables, ‘Yes’ indicates the extreme value was recalled, ‘No’ indicates that it was not.

  • Only recalled outcomes that matched the programmed values were included. i.e., if they saw the High Value Risky Door, 80 was logged as “Yes” and 40 was logged as “No.” If they saw the Low Value Risky Door, 0 was logged as “Yes”, 40 was logged as “No”. Participants who responded with any “other” values were removed from the analysis. Only Risky doors were evaluated.

  • In total, removing “other” responses resulted 15 participants being excluded from these analyses leaving a final N = 161.

  • \(\varphi_c\) is Cramér’s V, a measure of association for two nominal variables (i.e., an effect size of sorts). It can take a value between 0 and 1, with 0 indicating no association (i.e., full independence).

  • Post-hoc analysis on each \(\chi^2\) test was conducted by evaluating the standardized residuals from the chi-square analysis because it seemed the most intuitive approach. However, pairwise Pearson Chi-squared or Fisher Exact tests across the three conditions were also run. The p_value_adj column shows corrected p-values using the Holm-Bonferroni method.





Frequency Judgement

Plot - Means and 95% CIs



2x3 ANOVA using linear mixed-effects models fit by maximum likelihood


Main Effects

Model df AIC BIC logLik Test L.Ratio p.value BF_10
base 1 4 3240.858 3256.490 -1616.429 NA NA NA
cond_mod 2 6 3244.737 3268.185 -1616.368 1 vs 2 0.121 0.941 0.003
HL_mod 3 7 3221.010 3248.366 -1603.505 2 vs 3 25.727 0.000 20117.182
int_mod 4 9 3224.021 3259.194 -1603.010 3 vs 4 0.989 0.610 0.004

Main Effect of Group:

\(\chi^2(2) = 0.121\), \(p = 0.941\), \(BF_{10} < .01\)

Main Effect of Value (High vs. Low):

\(\chi^2(1) = 25.727\), \(p < 0.001\), \(BF_{10} > 150\)

Interaction:

\(\chi^2(2) = 0.989\), \(p = 0.61\), \(BF_{10} < .01\)


Planned Contrasts

Value Std.Error DF t-value p-value r_effect
(Intercept) 55.991 1.819 181 30.778 0.000 0.916
group 8020_vs_50 0.454 2.512 181 0.181 0.857 0.013
group 2080_vs_50 -0.403 2.531 181 -0.159 0.874 0.012
fj_value1 -6.026 1.714 181 -3.517 0.001 0.253
group 8020_vs_50 :fj_value1 2.252 2.366 181 0.952 0.342 0.071
group 2080_vs_50 :fj_value1 0.647 2.384 181 0.271 0.786 0.020

EX 80-20 vs. EX 50-50:

\(b = 0.454\), \(t(181) = 0.181\), \(p = 0.857\), \(r = 0.013\)

EX 20-80 vs. EX 50-50:

\(b = -0.403\), \(t(181) = -0.159\), \(p = 0.874\), \(r = 0.012\)

EX 80-20 vs. EX 50-50 | Value (High vs. Low):

\(b = 2.252\), \(t(181) = 0.952\), \(p = 0.342\), \(r = 0.071\)

EX 20-80 vs. EX 50-50 | Value (High vs. Low):

\(b = 0.647\), \(t(181) = 0.271\), \(p = 0.786\), \(r = 0.02\)


Notes
  • ANOVA was conducted only on judgements for the risky extreme values (0 and +80).

  • Bayes Factors are reported as Inverse Bayes Factors, this means larger values indicate greater support for the effect. Anything less than 1 should be considered as contributing no evidence.





Experiment 1b

Contingencies


Low Value Options
  • Fixed: 20 (Door 3)
  • Risky: 0 or 40 (Door 4)
High Value Options
  • Fixed: 60 (Door 1)
  • Risky: 40 or 80 (Door 2)
Trials
  • Six blocks of 80 trials.
  • Decision Trials: (fixed low vs risky low) & (fixed high vs risky high) - 24 per block (12 low, 12 high)
  • Single door trials: one door to click - 40 per block
  • Catch trials: high vs low - 16 per block
BEST 80-20
  • Block 1: 80% chance of best outcome on risky options.
  • Block 2: 50% of best outcome on risky options.
  • Block 3: 20% chance of best outcome on risky options.
  • Block 4-6: 50% of best outcome on risky options.
BEST 20-80
  • Block 1: 20% chance of best outcome on risky options.
  • Block 2: 50% of best outcome on risky options.
  • Block 3: 80% chance of best outcome on risky options.
  • Block 4-6: 50% of best outcome on risky options.
BEST 50-50 (control)
  • All blocks: 50% of best outcome on risky options.


Participant Information

Pre-catch exclusion

N = 190

group gender n age_mean age_sd age_median age_IQR
BEST 50-50 F 50 19.220 3.699 18.5 1
BEST 50-50 M 16 19.375 1.544 19.0 2
BEST 80-20 F 48 19.500 3.156 18.0 1
BEST 80-20 M 13 19.231 1.787 19.0 1
BEST 20-80 F 38 18.842 1.636 18.0 1
BEST 20-80 M 25 19.840 1.748 19.0 2


Post-catch exclusion

N = 183

group gender n age_mean age_sd age_median age_IQR
BEST 50-50 F 46 19.326 3.842 19 1
BEST 50-50 M 15 19.467 1.552 19 2
BEST 80-20 F 48 19.500 3.156 18 1
BEST 80-20 M 13 19.231 1.787 19 1
BEST 20-80 F 36 18.889 1.670 18 1
BEST 20-80 M 25 19.840 1.748 19 2


Condition totals:

group n age_mean age_sd age_median age_IQR
BEST 50-50 61 19.361 3.411 19 1
BEST 80-20 61 19.443 2.907 18 1
BEST 20-80 61 19.279 1.752 19 2


  • Total number of catch trial exclusions = 7.

  • Catch trial exclusions were based on block 5 and 6 performance.




Catch Trials

Plot - Block means with 95% CIs





Risk Preference Trials

Plot - Block means with 95% CIs



Plot - Interaction plot with 95% CIs (Block 5 & 6)



Main Effects

ANOVA Table

Model df AIC BIC logLik Test L.Ratio p.value BF_10
base_mod 1 4 247.353 262.964 -119.677 NA NA NA
grp_mod 2 6 249.708 273.124 -118.854 1 vs 2 1.645 0.439 6.000000e-03
value_mod 3 7 157.116 184.435 -71.558 2 vs 3 94.592 0.000 1.813658e+19
grp_val 4 9 153.345 188.469 -67.673 3 vs 4 7.771 0.021 1.330000e-01

Main Effect of Group:

\(\chi^2(2) = 1.65, p = 0.439, BF_{10} < .01\)

Main Effect of Choice Value (i.e., Extreme Outcome effect):

\(\chi^2(1) = 94.59, p < .001, BF_{10} > 150\)

Interaction: Group \(\times\) Choice Value:

\(\chi^2(2) = 7.77, p = 0.021, BF_{10} = 0.133\)

  • Note: while the p-value is technically significant, the Bayes Factor implies the interaction contributes nothing worthwhile.


Planned Comparisons

ANOVA Table

Value Std.Error DF t-value p-value sig r_effect
(Intercept) 0.383 0.033 180 11.747 0.000 TRUE 0.659
group 8020_vs_50 0.024 0.046 180 0.511 0.610 FALSE 0.038
group 2080_vs_50 0.058 0.046 180 1.267 0.207 FALSE 0.094
choice_value1 0.159 0.022 180 7.329 0.000 TRUE 0.479
group 8020_vs_50 :choice_value1 0.013 0.031 180 0.435 0.664 FALSE 0.032
group 2080_vs_50 :choice_value1 -0.067 0.031 180 -2.173 0.031 TRUE 0.160

BEST 80-20 vs. BEST 50-50:

\(b = 0.024, t(180) = 0.511, p = 0.61, r = 0.038\)

BEST 20-80 vs. BEST 50-50:

\(b = 0.058, t(180) = 1.267, p = 0.207, r = 0.094\)

BEST 80-20 vs. BEST 50-50 | Choice Value (High vs. Low):

\(b = 0.013, t(180) = 0.435, p = 0.664, r = 0.032\)

BEST 20-80 vs. BEST 50-50 | Choice Value (High vs. Low):

\(b = -0.067, t(180) = -2.173, p = 0.031, r = 0.16\)

Notes
  • All plots and analyses are showing the combined effect of block 5 and 6.
  • Bayes Factors are reported as Inverse Bayes Factors, this means larger values indicate greater support for the effect. Anything less than 1 should be considered as contributing no evidence.




First-Outcome Judgement

Plot - Proportions


Note:

  • Proportions are taken within group and context (High/Low). E.g., All three bars in the top left facet sum to 1, all three bars in the top middle facet sum to 1, and so on.

  • Since non-numeric responses were permissible by the experiment’s program, data was filtered to remove all non-numeric inputs. If a participant’s inputs had a cumulative sum less than 20 or greater than 300, their results were removed. In total, 6 participants met this criteria.


2x3 Chi-Squared Tests

High Value Results

High Value Contingency Table
No Yes
BEST 50-50 15 43
BEST 80-20 10 47
BEST 20-80 23 30


\(\chi^2(2) = 9.31, p = 0.01, \varphi_c = 0.24\)

  • Recalling the extreme value is dependent on condition.
Standardized Residuals
No Yes
BEST 50-50 -0.56 0.56
BEST 80-20 -2.27 2.27
BEST 20-80 2.89 -2.89
p-values
No Yes
BEST 50-50 0.572 0.572
BEST 80-20 0.023 0.023
BEST 20-80 0.004 0.004








The BEST 80-20 and BEST 20-80 conditions both differ significantly from the expected value. The BEST 80-20 condition tends to recall the extreme value, whereas BEST 20-80 does not.

  • BEST 80-20: \(z = 2.27, p = 0.023\)

  • BEST 20-80: \(z = -2.89, p = 0.004\)


Low Value Results

Low Value Contingency Table
No Yes
BEST 50-50 5 53
BEST 80-20 12 45
BEST 20-80 7 46


\(\chi^2(2) = 3.7, p = 0.157, \varphi_c = 0.15\)

  • Cannot rule out that recalling the extreme value is independent of group.
Standardized Residuals
No Yes
BEST 50-50 -1.524 1.524
BEST 80-20 1.796 -1.796
BEST 20-80 -0.271 0.271
p-values
No Yes
BEST 50-50 0.128 0.128
BEST 80-20 0.072 0.072
BEST 20-80 0.786 0.786








  • None of the three conditions differ significantly from the expected value.


Post-Hoc Pairwise Fisher-Exact Tests

value comparison odds_ratio p_value CI_low CI_upper p_value_adj
High BEST 50-50 : BEST 80-20 1.6325 0.3667 0.6105 4.5340 0.9612
High BEST 50-50 : BEST 20-80 0.4583 0.0713 0.1883 1.0884 0.3526
High BEST 80-20 : BEST 20-80 0.2809 0.0037 0.1037 0.7154 0.0223
Low BEST 50-50 : BEST 80-20 0.3569 0.0705 0.0914 1.1908 0.3526
Low BEST 50-50 : BEST 20-80 0.6226 0.5456 0.1453 2.4571 0.9612
Low BEST 80-20 : BEST 20-80 1.7435 0.3204 0.5713 5.7317 0.9612


  • Recalling the extreme value was found to be dependent on condition (\(p = 0.022\)) only when high value options in the BEST 80-20 and BEST 20-80 conditions are considered (Row 3 in the table). Everything else fails to reach significance.


Note:

  • In the contingency tables, ‘Yes’ indicates the extreme value was recalled, ‘No’ indicates that it was not.

  • Only recalled outcomes that matched the programmed values were included. i.e., if they saw the High Value Risky Door, 80 was logged as “Yes” and 40 was logged as “No.” If they saw the Low Value Risky Door, 0 was logged as “Yes”, 40 was logged as “No”. Participants who responded with any “other” values were removed from the analysis. Only Risky doors were evaluated.

  • In total, removing “other” responses resulted 9 participants being excluded from these analyses leaving a final N = 168.

  • \(\varphi_c\) is Cramér’s V, a measure of association for two nominal variables (i.e., an effect size of sorts). It can take a value between 0 and 1, with 0 indicating no association (i.e., full independence).

  • Post-hoc analysis on each \(\chi^2\) test was conducted by evaluating the standardized residuals from the chi-square analysis because it seemed the most intuitive approach. However, pairwise Pearson Chi-squared or Fisher Exact tests across the three conditions were also run. The p_value_adj column shows corrected p-values using the Holm-Bonferroni method.





Frequency Judgement

Plot - Means and 95% CIs



2x3 ANOVA using linear mixed-effects models fit by maximum likelihood


Main Effects

Model df AIC BIC logLik Test L.Ratio p.value BF_10
base 1 4 3215.285 3230.896 -1603.643 NA NA NA
cond_mod 2 6 3211.195 3234.611 -1599.598 1 vs 2 8.090 0.018 1.560000e-01
HL_mod 3 7 3145.344 3172.662 -1565.672 2 vs 3 67.851 0.000 2.831444e+13
int_mod 4 9 3148.096 3183.220 -1565.048 3 vs 4 1.248 0.536 5.000000e-03

Main Effect of Condition:

\(\chi^2(2) = 8.09\), \(p = 0.018\), \(BF_{10} = 0.156\)

Main Effect of Value (High vs. Low):

\(\chi^2(1) = 67.851\), \(p < 0.001\), \(BF_{10} > 150\)

Interaction:

\(\chi^2(2) = 1.248\), \(p = 0.536\), \(BF_{10} < .01\)


Planned Contrasts

Value Std.Error DF t-value p-value r_effect
(Intercept) 59.779 1.589 180 37.609 0.000 0.942
group 8020_vs_50 -2.648 2.248 180 -1.178 0.240 0.087
group 2080_vs_50 -6.943 2.248 180 -3.089 0.002 0.224
fj_value1 -7.844 1.589 180 -4.935 0.000 0.345
group 8020_vs_50 :fj_value1 1.205 2.248 180 0.536 0.593 0.040
group 2080_vs_50 :fj_value1 -1.287 2.248 180 -0.572 0.568 0.043

BEST 80-20 vs. BEST 50-50:

\(b = -2.648\), \(t(180) = -1.178\), \(p = 0.24\), \(r = 0.087\)

BEST 20-80 vs. BEST 50-50:

\(b = -6.943\), \(t(180) = -3.089\), \(p = 0.002\), \(r = 0.224\)

BEST 80-20 vs. BEST 50-50 | Value (High vs. Low):

\(b = 1.205\), \(t(180) = 0.536\), \(p = 0.593\), \(r = 0.04\)

BEST 20-80 vs. BEST 50-50 | Value (High vs. Low):

\(b = -1.287\), \(t(180) = -0.572\), \(p = 0.568\), \(r = 0.043\)


Notes
  • ANOVA was conducted only on judgements for the risky extreme values (0 and +80).

  • Bayes Factors are reported as Inverse Bayes Factors, this means larger values indicate greater support for the effect. Anything less than 1 should be considered as contributing no evidence.





R Citations

R version 4.4.2 (2024-10-31 ucrt)

  • ggh4x (0.2.8) (van den Brand, 2024)
  • Hmisc (5.1.3) (Harrell Jr, 2024)
  • nlme (3.1.166) (Pinheiro et al., 2023)
  • RColorBrewer (1.1.3) (Neuwirth, 2022)
  • rcompanion (2.4.36) (Mangiafico, 2024)
  • tidyverse (2.0.0) (Wickham et al., 2019)




References

Harrell Jr, F. E. (2024). Hmisc: Harrell miscellaneous. https://CRAN.R-project.org/package=Hmisc
Mangiafico, S. S. (2024). rcompanion: Functions to support extension education program evaluation. Rutgers Cooperative Extension. https://CRAN.R-project.org/package=rcompanion/
Neuwirth, E. (2022). RColorBrewer: ColorBrewer palettes. https://CRAN.R-project.org/package=RColorBrewer
Pinheiro, J., Bates, D., & R Core Team. (2023). Nlme: Linear and nonlinear mixed effects models. https://CRAN.R-project.org/package=nlme
R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
van den Brand, T. (2024). ggh4x: Hacks for ’ggplot2’. https://CRAN.R-project.org/package=ggh4x
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686